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ABSTRACT 

Findings from a study that explored the ways in which 
principals use information on school quality are presented in this 
paper. The effects of school level, years of administrative 
experience, students' socioeconomic status, and school performance 
are also examined. Methodology involved interviews ard follow-up 
interviews with a total of 38 elementary, middle, and high school 
principals from 12 California school districts. Findings indicate 
that the most frequently used information source is school/classroom 
observation, and that test data are most frequently used to 
communicate with teachers and parents and to identify areas of 
instructional strengths and weaknesses. Recommendations are made for 
improving principal analysis of school data and communication with 
the public. Appendices contain the interview protocol, comment sheet, 
and survey data (8 graphs and 5 tables) . (LMI) 
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In today's information age school principals are bombarded with enormous 
amounts of inrormation about those whom they serve. Test scores, attendance rates, 
discipline records, teacher, parent, and student input, and observations aie just a 
few of the sources of Information about students that principals receive. Given so 
many alternatives, which ones do principals really use and for what purpose? Does 
one's preferences for different sources change according to how many years 
experience one has as a principal, whether one Is an elementary, middle, or high 
school principal, or whether one's students are from low, middle, or high socio- 
economic standings? How can those of us interested In promoting the systematic 
use of data In schools help principals to use data more effectively? In this paper, we 
will begin to answer these questions and provide suggestions for improving the 
reporting of school data. 

The Multi-Level Evaluation System (MLES) project, funded by the Office of 
Educational Research and Improvement (OERI) and administered by the UCLA 
Center for Research on Evaluation, Standards, and Student Testing (CRESST), has 
studied the use of school information for almost three years. The MLES project is 
investigating the feasibility of developing comprehensive information systems that 
will serve the planning and policy needs of school-based educators, district 
administrators, and school boards and intends to develop a set of design 
specifications for such systems. The first stage of the project was to compile a multi- 
disciplinary literature review that summarized the guiding principles for the design 
of school-based management information systems and the presentation of school 
data reports. Stage two featured a review of existing district reporting practices. 
This paper discusses some of the results of stage three, an interview study of 73 
principals, school board members, and other district administrators about how they 
process and use information on school quality. 

We conducted a study that asked principals to define how they use school 
data and other information. We interviewed the principals using open-ended 
questions and carried out a content analysis of their responses. The procedures used 
in the interview and content analysis stages are desaibed in the methodology 
section of this paper. Principals generated a long list of Information sources and 
some patterns of usage emerged. Some ways in which they misused information also 
appeared. The findings of the content analysis are discussed in the results section of 
this paper. The results of this study hold valuable lessons for those who create 
school data reports. The implication and conclusion section of this paper offers 
applications which can increase the utility and clarity of school data reports. 



Methodology 

The MLES project's multidisdpUnary literature review and examination of 
district reporting practices were the basis for hypotheses about how principals use 
data in the real world and the variables which are likely to influence such use. 
These were the basis for the sample design and the Interview protocol. 

The Interview Protocol 

The Interview protocol was developed to examine questions and variables of 
interest and then revised based on field test results. The final version had three 
parts: 

Pre-intervlr v. The pre-interview consisted of an Introduction, a statement 
of purpose, a description of the general content of the interview, an assurance of 
anonymity, permission to tape the interview, and solicitation of clarifying questions. 
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Interview. The second part of the protocol contained the interview 
questions. The questions concerned the sources, aiterla and application of 
information principals use to assess the quality of education in their schools. Other 
questions were about principals' interest in sub-groups test data, preferences for 
particular formats for test reports and additionally desired information. The final 
question aslced principals how much they felt test data reflects what is important in 
schooling. Copies of the questions used are located In Appendix A. 

Post-interview. The final section of the protocol consisted of questions for 
the interviewer (which were completed as soon as possible after the interview). 
This sheet had five comment areas: key themes, areas of concern, areas of 
confusion, personal reflections and suggestioris for future interviews (see Appendix 
B). 

The Subject Population 

Twelve districts were selected for participation in the study to represent a 
range of socio-economic status levels, diversity with regard to ethnic composition, 
and a range of siz»s. Within each district, three principals and three board members 
were to be randomly selected for interviews. 

This paper focuses on the subsample of 38 school principals from all 12 
districts which were part of the study. Of the 12 school districts, two districts had 
less than 10 schools; eight had ten to 30 schools (classified as medium), and two had 
more than 30 schools (classified as large). The schools served in this obtained sample 
varied in ethnic composition and sodo-economic level. The ethnic composirion of 
most schools was either mixed-minority majority, mixed white majority or majority 
Latino schools. Other minorities represented were Afro-American and Asian. 

Three school levels were represented: eighteen elementary school 
principals, eight middle school and eleven high school principals. Job experience 
data on twenty-sbc of the principals indicated that six were beginners with one to 
two years experience, eight had three to sbt years experience, and twelve were 
veterans with seven or more years. 

Interview Procedures 

Staff from the UCLA Center for Research on Evaluation, Standards, and 
Student Testing (CRESST) and directors of district research and evaluation offices 
were reauited and trained in study procedures. Training included discussion and 
practice in the use of the interview protocol as well as directions for randomly 
selecting respondents, and for taping, summarizing and returning completed 
interviews. Interviewers then used the protocol to obtain data. 

Each interviewee was flrst contacted by phone and asked to be involved in 
the study. Most Interviews took approximately one half hour. All the interviews 
were tape recorded. The interviews were conducted between May and September, 
1989. 

Interviewers summarized the results of their interviews as soon as possible 
after each interview. The interview summaries and tapes were turned over to the 
MLES staff. The MLES staff then coded and analyzed the results. 

Analysis of the Data 

Tape interviews were summarized and a code book was assembled and tested 
by four coders. Formal coding did not begin until there was one hundred percent 
agreement on eight test interviews. For the study reported in this paper frequencies 



were tabulated and analyzed for both overall principal data and on the basis of four 
specific contextual variables: the school's ethnldty and the school's SES, the 
principal's years of experience, and the school's performance on CAP reading scores. 

Limitations of the Data 

Like much of research conducted In natural settings, this study had 
difficulties In controlling the presence of contextual variables of potential interest. 
In addition, sample sizes limit the generalizabllity of findings, particularly for those 
analyses which attempt to assess the effects of contextual variables (e.g. principal's 
years of experience, school SES, and school CAP achievement). We also were not 
able to separate out the effects of school level, principal's experience, school SES, 
and school CAP achievement from each other. Due to these caveats, the findings 
presented in our results section should be viewed as preliminary patterns and areas 
for future research. 



Results 

What Information Do Principals Use? 

Sources of information used. School principals who participated in our 
study use a total of twenty-four sources of information to judge the quality of 
education in their schools. The sources are both quantitative and qualitative. The 
median number of sources used by principals is eight. 

Our study found that the most frequently used source is school and classroom 
observation (nearly 87 percent utilize observations). Many principals found 
classroom observations to be their most valuable source because observations provide 
immediate feedback as well as give the principals a holistic and personal sense of 
how the school is functioning. For example, one elementary school principal told 
us that classroom observations are his most valuable source because they provide 
"first hand information" that he can gather every day. Another principal said, a 
classroom observation "...tells me more than anything else, so I visit at least five 
classrooms daily." 

Other widely used sources included CAP scores, mentioned by seventy- four 
percent of the principals, and other norm-referenced tests mentioned by seventy- 
one percent. Although many principals use test scores when evaluating their 
schools, they feel that these test scores receive more attention than they deserve 
because of the emphasis placed on them by state and district officials. Principals 
expressed their frustration about this over-emphasis of test scores. According to one 
principal, "Test scores are rated as important because of the State and District's 
emphasis on them." Another complained that, "With the over-emphasis of testing 
by bureauaats, testing becomes an end, rather than a means to an end." Further, 
test scores seem to have their biggest impact as a source when they first are 
reported. A Junior high school principal who mentioned test data as his first source 
of school information also said: 

Down the line, test data is not really at the top (important). It's only at the 
top when it's in the paper. When it's in the minds of people; other than 
that, it's not a high priority. People forget about it. 

CAP scores were also the most controversial source, with sixteen percent of 
the principals saying that they have no value as a source of information. The 
principals complained about CAP'S instructions, self-report measures, and reporting 
of results in comparison bands. (These bands are supposed to compare schools 
similar in socio-economic standing.) A high school principal who preferred CTBS to 



CAP told us, The CAP test contains too many self-report items with bad and 
ambiguous instructions. Also, the CAP comparison bands are unfair because they do 
not take Into account bussing.' 

Informal teacher, parent and student input were mentioned as important 
sources by almost half of the principals and ranked fourth, fifth and sixth 
respectively among these mentioned. Student grades ranked a close seventh. 
Other quantitative sources such as teacher turnover rates, drop-out rates and 
mobility rates were among the least used. Thus, while quantitative measures are 
viewed as important— ranking second and third in our study— qualitative indicators 
appear to receive more overall attention than quantitative indicators. (Detailed 
results of our analysis are summarized in Table 1 and Graph 1 in Appendix C.) 

Effect of school level. The data yielded interesting contrasts, depending 
upon grade level. There is a marked difference in what elementary and high school 
principals consider important sources of information. High school and middle school 
principals rely more heavily on informal teacher, student and parent input than do 
elementary school principals. Another difference is that elementary and middle 
school principals are more interested in CAP and other achievement tests while 
high school principals are more interested in college preparation exams such as the 
SAT. In addition, the use of classroom observation declines as the level of the school 
increases (see Table 2). 

Effect of years of experience. Analyses by experience level based on a 
sub-sample indicate that those with six or more years of experience appear to use 
fewer sources of school information. Veterans of six or more years used a median of 
six sources while beginning principals with two or less years of experience used a 
median of nine sources. In addition, veterans appear to rely less on CAP and norm- 
referenced test data than do beginning principals. Furthermore, veteran principals 
are less likely to seek or use the advice of superintendents or other district 
administrators (such as research and evaluation directors). Additional research would 
be necessary to validate these findings given the small sample size (refer to Table 3). 
A difference in the emphasis placed on test data and district advice can be seen in 
these descriptions of the use of test data offered by a beginning and veteran 
elementary school principal: 

(Beginner) I rely on workshops given by the different project heads, such as 
the CTBS or Director of Federal Projects, to get an understanding of the 
district data and suggestions on how to present the data to my staff. 

(Veteran) I tend to be skeptical about test data.. .its just one piece of 
information about a child. 

Effect of SES and CAP achievement The two contextual variables of socio- 
economic standing of the student population and the school's performance on the 
reading section of the CAP exam were both related similarly to the use of sources. 
This was probably a function of a high correlation between SES and success on the 
CAP exam. Four differences in the use of data emerged between principals from 
schools of different economic standings and levels of achievement on the CAP 
exam. Principals of schools with higher SES levels and which perform above the 
state average on CAP exams tend to be more interested in student success at the 
next grade or school level, attend more to informal parent and student input, and 
seek less guidance from district administrators such as research and evaluation 
directors than do principals of schools of lower SES students and those whose 
students perform below the state average on CAP. The lack of interest in 
administrator input among the high SES and achieving school principals may be due 
to the common tendency to not search out more information when feedback is good 
and supports prior expectations (see Tables 4 and 5). 
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The differences In school evaluation approaches between low and high 
socio*econoinlc elementary school principals are illustrated in the quotes below. 
The principal of a low-income based school that has been at the lowest percentile 
per grade level based on national norms for years reported: 

To tell if my school is doing a good job, I look first at test scores, since they 
are widely publicized. I want to see whether my students are on grade level 
and whether there has been growth over the year. I feel testing is the most 
important objective source for determining student progress... My school has 
remained at about the same level test-%vlse for about five years. I consider 
this a plus in light of turnover in staff and the influx of new students. 

The principal of a high-income based school that consistently scores above the 90th 
percentile on standardized tests reported to us that to tell if her school is doing a 
good job: 

...I observe classrooms. I want to know that the kids are learning and how 
they feel. I go into classrooms and look for interaction, active learning, and 
interest on the part of the students. I also receive parent feedback on the 
annual survey and from parents involved on the school site advisory 
committee. I also use test scores and find out how well students are doing in 
intermediate school. 

What Criteria Do Principals Use When Making Their Assessments? 

Observing schools or classrooms. The most popular source of information 
named by principals was personal observations on the school and classroom levels. 
Most principals, about two-thirds of the respondents, had specific areas of interest 
in mind when they made their observations. School or classroom climate was the 
most popular area of interest and was defined in terms of student-student and 
student-teacher interactions. When asked what he would find In a classroom with a 
positive climate, one principal named, "the teacher interacting positively with the 
children; a child who does something wrong is not put down — rather he is 
encouraged; and positive exchanges among the kids." 

Besides interactions among students and teachers, positive classroom climate 
was also defined in terms of the students' attitudes toward learning and the teachers' 
attitudes toward teaching. Students were seen as experiencing a positive classroom 
atmosphere 'if students are happy, meaningfully involved, and like and understand 
what they are doing.' Teachers were seen as creating a positive classroom 
atmosphere when they displayed 'an attitude that education is important, that they 
love kids, that they want to impart this information.' 

Other frequently mentioned areas of interest when observing a classroom 
were instructional content, teaching methods, student involvement in instruction, 
student work products, and teacher morale. (See Graph 2 for actual percentages of 
factors examined.) 

Analyzing test daU. Standardized tests such as CAP, CTBS, and others were 
the second anJ third most used sources of school information. One third of the 
principals interviewed reported needing help to interpret test results. Many more 
principals expressed that the time necessary to analyze test data thoroughly is 
limited and inconvenient. A junior hi^ school principal explained: 

We're (principals) flooded with data. We are right now so caught up in the 
day-to-day management of our schools, we don't get to the point of analyzing 
data... The direction we're going means that thfs kind of data (test data) is 



going to mean less and less to us unless somebody comes along and pulls it 
out for us, condenses It, and summarizes it. 



For those principals who read and interpreted their own test results, their 
primary strategy was to look for trends of performance over several years (74 
percent said they employed this strategy). Half of the principals said they 
compared their school's performance to other similar schools. As one principal told 
us, "I look at average scores to see how my school compares with other schools, 
particularly schools that serve similar types of students. I also think that comparing 
stuUent progress over the years — cohort tracking— is important." 

Also common when interpreting test results was comparing their school's 
performance to the national average or some other grade equivalency score. The 
desire to not fall below grade level on the national norm was strongly expressed. 
One principal told us, "We always strive and work toward being on grade level- 
always!" Besides the goal of being at or above grade level, 42 percent of the 
principals said they approached the data reports with predetermined goals for their 
schools, such as to be above the 75th percentile (see Graph 3). 

How Do Principals Apply Test Data? 

Using data. In addition to how principals make sense of test data, we were 
Interested in how principals use and apply it. We found that the most common use 
of test data is to communicate it to teachers and parents (three-quarters of the 
principals reported doing so). Also popular was using test data to identify areas of 
instructional strength and weakness. Slightly less frequent was follow-up to actual 
use of this data for instructional reforms. One principal told us that when she 
receives test results from the District: 

I return to my school and meet with the faculty all together and by grade 
level. In the grade level meetings we compare for each subject area teachers' 
scores with others, their strengths are recognized and areas in need of 
greater emphasis are identified. Ways to ameliorate a particular problem are 
discussed in 
detail. 

Over a third of the principals use the test data to answer spedfic 
programmatic questions. For Instance, a principal may specifically check the 
movement of the third grade's CAP reading score because he or she is concerned 
about the introduction of a new basal reader in the second grade. Finally, almost a 
quarter of the principals mentioned that they look for aberrations and unusual 
results that may highli^t potential problem areas (see Graph 4). 

Discrepancy handling. In order to determine to what extent principals 
rely upon test data and whether they use it unaitically, principals were asked how 
they respond when faced with disaepandes between test data and other sources of 
data such as observations or input from teachers. Their answers suggest that 
prindpals are fairly sophisticated in their use of test data. Over one quarter of the 
prlndpals said that they would consider multiple sources before determirUng the 
meaning of the discrepancy. When making sense of inconsistent information, one 
prindpal told us how he involves his staff in considering these multiple sources of 
information. He said: 

The only solution to that (a disaepancy) is to analyze what the test Is 
measuring and what is the population. We do this through a teacher 
committee and dedde what information to pay attention to. 
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We believe that this first approach to handling discrepancies, considering 
multiple sources, Is the most appropriate and should therefore be further 
encouraged through In-servlce and pre-servlce trainings of school administrators. 

Another quarter of the principals said In the face of a discrepancy, they 
would question the test. Other strategies mentioned by the principals were to 
question the match between the test's coverage and the classroom instructional 
coverage or to look to alternative explanations such as unusual weather or high rates 
of student mobility. This is what a principal who questioned the test as well as 
considered alternative explanations had to say: 

If there is a discrepancy between test scores and classroom observations, I 
would look at changes in our student population, especially In the number of 
LEP students. I would trust my observations over the test scores. 

Only one quarter of the principals said they would believe the test in the 
face of their own or others' observations (see Graph 5). For these principals, the 
test results always had value, and in this case reflected some defect in the 
Instructional program. One such principal told us: 

We (the stafO have to go back and look and say. What's going on here? 
Why do we have this? I don't think you can ignore this (the test results). 
No matter what you think of the test; no matter how invalid you may think 
the test is; how prejudiced it is, you are still competing against yourself on 
whether you go up or down. 

It Is interesting to note that the principals' reliance on and interpretation of 
test findings does not appear to vary with the school's economic standing, the years 
of experience at their job, the grade levels present at their school, or how well their 
school does on standardized tests. Therefore their handling of data discrepancies 
involving tests does not appear to be a defensive reaction. 

Are Principals Interested in Sub-Group Differences? 

Interest in sub-group differences. Asked if they had any Interest in sub- 
group differences, almost half the principals expressed an Interest in performance 
differences among ethnic groups (primarily Latino, Afro-American, and White). 
Another sub-group of interest was those students who are limited in English 
proficiency Other bases for sub-groups of interest were socio-economic standing, 
gender, and levels of achievement, One tenth of the principals expressed the 
concern that sub-group information may lead to misuse and Inappropriate 
stereotyping. One principal who expressed concern about reporting sub-group 
differences told us, "I don't believe we should pigeon-hole kids or pit this group 
against the other. Instead, we must look at lots of variables like home input and 
quality of teachers in that school." 

What More Do Principals Want from School Reports? 

AddiUonally desired information. Most of the principals seemed reluctant 
to ask for any more information than they already have or receive. Of those who 
were Interested in additional information, the greatest Interest was in analyses 
showing relationships between test results and specific Instructional programs. 
These principals want to be able to tie test data back to their choice of textbooks, 
teaching methods, or other instructional reforms. Other desired information 
included: more data about individual students that could help with instructional 
diagnosis; more descriptive information about what the test is measuring; 
presalptlve information that tells the principal what instructional practices can be 
taken to rectify low scores in a particular area; information about other quantitative 
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indicators such as dropout rates, mobility rates, and attendance rates; and analyses 
between test scores and student demographics (somewhat the same as sub-group 
differences). Very few principals were interested in additional information about 
school climate, although this was a great area of concern when discussing classroom 
observations (see Graph 7). 

Preferred report formats. Principals reported their favorite format of 
presentation to be the graph, a finding which dosely relates to their overwhelming 
Interest In trends. This makes sense, because graphs are the dearest way to display 
trends. Far fewer prindpals like narratives and tables. Other format preferences 
expressed were to Include less technical Information, to provide an executive 
summary with reports, and to Improve the quality of keys for tables and graphs (see 
Graph 8). 



Implications and Conclusions 

This study set out to answer two basic questions: how do prindpals use 
school data and how can we help them to use that data more easily and accurately. 
This sedlon of the paper addresses the latter question; It offers some 
recommendations for those people Involved In the creation of school Information 
reports. The recommendations are broken down into two major categories: helping 
prindpals better analyze school data and helping prindpals better communicate 
with the public about the quality of their schools. 

Helping Principals To Analyze School Data 

Variety of sources. The first way In which one might help or ndpals 
analyze school data Is to make them aware of the full range of quantitative and 
qualitative information available to them. In our study prindpals Identified 24 
sources of Information which at least one prlndpal, and usually more, found useful. 
Making prindpals aware of the full array may lead them to Incorporate new sources 
of information Into their own judgements about school quality as well as maice them 
aware of sources that may be used to more conaetely show what they already 
Inherently believe about their schools. 

Evaluating the quality of test data. Most principals realize that test data 
represents only an estimation of how well their students are doing and that 
sometimes these estimations may be erroneous. However, a sizable minority of the 
prindpals interviewed In this study do not appear to be aitical users of test data. 
Fully one-quarter accept without question the results of tests, even in the face of 
other evidence to the contrary. It would be benefidal for all prindpals to approach 
test results with skeptldsm and be fully familiar with their potential shortcomings 
(such as a poor test, poor testing environment, mismatch of test and instructional 
goals). Similarly, a number of prindpals appear to need assistance In more 
effectively using test results to improve their Instructional programs (e.g., in 
identifying areas of instrudional strengths and weakness and in designing and 
following up on prog.jm changes to address those weaknesses). 

Relating test data to instruction. Principals want help relating test data to 
instrudional practices. When asked what additional information they wanted, 
nearly one-third of the prindpals wanted analyses that could relate data to 
instructional reforms and/or other changes in curriculum and Instrurtlon. A similar 
number wanted explldt recommendations about what instrudional changes should 
take place based on the test results. 

While both groups want help in better integrating testing with Instrudion, a 
distindion can be made l)etween these two groups. Those prindpals who want to 
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examine relationships between test results and instructional practices recognize that 
data can be used as a tool for decision-making. These principals appeared to have 
specific, albeit idiosynaatic, questions that they wanted the test data to help 
answer, e.g., has the new science textbook improved achievement? Have changes 
in grouping practices affected the performance of higher achieving students. This 
group of principals seems comfortable with using data and might benefit from 
automated, analytic tools to help them answer their questions. 

The second group of principals, however, appear to be looking for simple 
answers to complex professional problems. They want prescriptions—'just tell me 
what to do" appears to be their sentiment. This second group of principals might 
benefit from learning how to approach test data with specific questions that could 
help them in analyzing specific areas of instruction. For both groups, however. It 
would be wise to reinforce the Ideas that large instructional decisions should not be 
based on test data alone. 

Formats. The choice of formats used in a school report can influence the 
degree to which principals can easily analyze and understand data. In this study, 
principals dearly state a preference for test data presented in terms of trends, and 
further report a strong preference for seeing those trends in graphical formats. 

Keys presented with graphs apparently are a source of trouble. Keys that 
accompany graphs should be able to stand alone and be easily understood. In some 
cases a brief statement summarizing the trend's interpretation also would be helpful. 
Finally, executive summaries that present key trends, identify aberrations, areas of 
strengths and weaknesses, and compare school results to other similar schools would 
prove helpful in many cases. 

Helping Principals to Communicate School Data tc the Public 

Reporting to the public. Principals overwhelmingly use and value school 
and classroom observations more than test data in assessing the quality of their 
schools. They balance the use of qualitative and quantitative data. Then why 
should they use test data as the number one, and often only, indicator of school 
quality when reporting to the public? The problem is that many principals do not 
know how to report qualitative data to the public in a way that is aedible, 
defensible, and clearly understood. There are also issues about the objectivity of 
such observations. Developing observational protocols that can be more easily 
summarized as well as quantified might be a solution. Such protocols could 
operationalize each of the areas of concern discussed in this paper, incorporating 
the research base on these areas, and perhaps result in a bank of observation items. 
A principal could then select those areas of most concern to him or herself and 
develop a protocol on that basis. With the help of a research and evaluation 
director, the results of such a protocol could be summarized and quantified for the 
public. The public— and parents — also could be made privy to important elements 
assessed by the protocol so that they could make their own observations. 

Besides making observational information more credible, principals need 
professional yet easy ways in which to gather, summarize, and report student, 
parent, and teacher feedbadc, perhaps through routine questionnaires or random 
phone interviews. Current attempts at school report cards will test the feasibility 
and validity of such practices. 

Concluding Remarks 

In conclusion, this study ij an attempt to understand the use of data by 
school principals. Although it was based on a small sample size, the study suggests 
patterns of information usage that can be supported, expanded, and/or changed 
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through specific action. The Implications of this study can be applied Immediately 
to the construction of school reports and the training of school administrators. We 
hope that this paper also will spur future theoretical studies of school information 
use and prove to be of practical help out In the real world of educational evaluation. 
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Appendix A 



CRESST Multilevel Evaluation Systems Project 
INTERVIEW PROTOCOL 

for School Board Members and School Principals 

May 9. 1989 



1. "Briefly, how do you know how good a Job your school(t) are doing 
for students?" (List sources of Information mentioned, e.g., parent 
phone calls, newspaper articles/editorials, personal observation of 
schools, test scores, etc.) 

a. For each source mentbned, probe for how influential that source is in their 
judgment of their school(s) and why it carries that weight. ('You mentioned a 
number of sources of information, I'd like to know a little more about how much 

importance you place in that source and why." How about . Would you say 

it's of overpowering importance, important, or only somewhat important? Why) 

b. If test scores are not initially mentioned, probe: 'Do you use any test data?' (If 
no. probe for why not; if yes, probe for which specific ones are used) 

2 . I'd like to know a little more about how you use this information to 
Judge your school(s). 

a. For each source mentioned as important above, ask: 

If your schools were doing a good job, what would you expect to see in [the 
information source]?' 

What in [the information 6c jrce] signals to you that there is a problem or that 
some change is needed in your school/ district? 

b. Suppose there's some discrepancy between these various sources of 
information. For example, suppose you thought your math program was pretty 
good, but your math test scores are relatively low, what wouki you do/think? 

3 . (Show district/school testing report) "There's a lot of information 
in reports like this and not everyone who reads these reports goes about 

it In the same way. When you get a report like this, how do you attack it 
to make some sense out of it? What's the process you go through to find 
out what you want to know? 

(probe if necessary with questions such as: what's the first thing that gets your 

attentton? And then what. Is there anything else that 

particularly draws your attention? What questions are you asking yourself as 
you review such reports?) 

4. When you look at information such as this, are you interested in 
knowing how different groups of students within the district perform? 
(e.g., how LEP students perform, how Hispanic, Black, Asian, Caucasians 
perform, how girls vs. boys perform?) 
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a. ffyos, which subgroups are of most interest to you? 



b. If yes, ti2M is this Information useful to you? 

5. Let's think .« little more about the Information that'a presented In 
school/ district testing and evaluation reports. We're interested In 
knowing how to do this better. 

a. First, about how the information is presented. Reports like this typically 
include narrative text, data tables, graphs and the like. What's the easiest way 
for you to get information? 

Is there soms part of this report that is particularly effective? 
What makes it effective? 

(Probe for format comments, e.g., are there any displays that you find 
particularly in formative ?) 

b. Second, about what information is presented. Is there additional information 
about students, schools, or communities that you could like to see included? Or 
things ihat are here that you would just as soon see deleted? 

6. Would you say that test scores capture most of what's important in 
schooling? [probe for whether it represents most of what's important in 
students academic achievement.] 

7. Any ot^er comments or suggestions? 
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Appendix B 



IS 



1) Suaunary of the Informant's personal approach to school evaluation. 
(Use your own words to describe how you perceive the informant's actions 
and attitudes towards school evaluation.) 



2) Topic areas of concern for the informant (burning issues mentioned that 
relate or don't relate to the questions asked) . 



3) Problems or confusion (the informant's lack of understanding of 
specific questions or your confusion about the informant's answer for a 
specific question) . 



4) Personal reflections on the interview and suggestions for future 
interviews. 



5) Emotional tone of interview (your feelings and your perception of the 
informant's feelings). 



6) liocation: 
Time: 

Environment (physical and social setting) : 

ERIC 1 9 



Appendix C 



20 



• 


Table 1: Sources 


of Information 


Used 






by Principals 






m 




ti uiscounted 


% Used 


Rank 




Sch/Class Observation 


3 


87 


1 




CAP 


16 


74 


2 




Norm Reference Tests 


8 


71 


3 


m 




*> 
J 


58 


4 




Informal Parent Input 


8 


53 


5 




Informal Student Input 


3 


47 


6 




Grades 




47 


6 




Formal Teacher Input 




34 


8 


m 




3 


21 


9 




Discipline 




24 


10 




Formal Student Input 


- 


21 


11 




Community Input 


3 


21 


11 




Super int/Other Adm Input 




21 


11 


m 


■ • W A IvICi A IwC X\CI 


D 


19 


14 




Proficiency Tests 




19 


14 




Newspapers 


5 


18 


16 




Success - Next Level 




18 


16 




College Prep Exams 




18 


16 


m 


Entries - Higher Level 




13 


19 




Formal Parent Input 




13 


19 




Mobility Rate 




11 


21 




Teacher Turnover 




8 


22 




Dropout Rate 




8 


22 


m 


Awards 




8 


22 
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FYequen(\v of Cilalion of Sources 

Scliool F^riiicipals 



Sch/Closs Obs 
CAP 
Norm Ref 
Inf Teach hp 
Inf Pgr Input 
hf St InfXJt 
Stu Grades 
I of7)i leach Inp 
Counselors 
Disci}:)line 
Commun Input 
f am St Input 
^^llpef/Dist Adm 
Attmdanre 
Prof Tests 
'4i(~( ess Nxt Lv 
Newsfxxiers 
(Joileqc' Pre{) 
t ntry I li fd 
' oirn P()i ln()ut 
Mobility 
A well (Is 
li 1 urn (jvrt 

( )f( (XJt lv 



22 
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23 



0 



^0 



.0 



8( 
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Table 2: 



Percentages of Principals who Use Sources 
By School Level 



Sources 



Elementary 



n 



18 



% 



Middle 
n = 8 
% 



High 
n = 11 
% 



Sch/Class Observation 
CAP 

Norm Reference Tests 
Informal Teacher Input 
Informal Parent Input 

Informal student Input 
Grades 

Formal Teacher Input 
Counselor input 
Discipline 

Formal Student Input 
Community Input 
Superint/Other Adm input 
Attendance Rate 
Proficiency Tests 

Newspapers 

Success - Next Level 
College Prep Exams 
Entries - Higher Level 
Formal Parent Input 

Mobility Rate 
Teacher Turnover 
Dropout Rate 
Awards 



94 


88 


63 


50 


100 


36 


8 


O J 


55 


39 


/D 


73 


50 


3U 


64 


28 


75 


55 


44 


50 


55 


39 


25 


27 


11 


50 


27 


22 


50 


9 


17 


13 


36 


17 


25 


18 


17 


25 


18 


17 


25 


18 


17 


13 


27 


11 


13 


27 


11 


13 


36 


0 


0 


64 


0 


0 


46 


11 


13 


18 


17 


13 


0 


11 


0 


9 


6 


0 


18 


6 


13 


9 
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Table 3: Percentages of Principals Who Use Sources 

By Experience at Job 



sources Beginner Intermediate Veteran 

n = 6 n = 8 n = i2 

% % % 



Sch/Class Observation 
CAP 

Norm Reference Tests 
Informal Teacher input 
Informal Parent Input 

Informal Student Input 
Grades 

Formal Teacher Input 
Counselor Input 
Discipline 

Formal student Input 
Community Input 
Superint/Other Adm input 
Attendance Rate 
Proficiency Tests 

Newspapers 

Success - Next Level 
college Prep Exams 
Entries - Higher Level 
Formal Parent Input 

Mobility Rate 
Teacher Turnover 
Dropout Rate 
Awards 



83 


75 


58 


88 


75 


58 


83 


88 


67 


67 


50 


50 


50 


63 


58 


67 


25 


58 


67 


38 


50 


0 


38 


25 


50 


25 


33 


33 


25 


42 


17 


13 


25 


50 


13 


17 


50 


38 


17 


17 


25 


25 


17 


25 


8 


33 


25 


8 


17 


25 


8 


33 


0 


16 


17 


13 


0 


0 


25 


8 


0 


13 


8 


17 


13 


0 


17 


13 


8 


17 


13 


8 
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Table 4: Percentages of Principals who use Sources 

By School SES 



Sources Low Middle High 

n«6 n«i4 n*4 

% % % 



Sch/Class Observation 
CAP 

Norm Reference Tests 
Informal Teacher Input 
Informal Parent input 

Informal student Input 
Grades 

Formal Teacher input 
Counselor Input 
Discipline 

Formal Student input 
Community input 
Superint/Other Adm Input 
Attendance Rate 
Proficiency Tests 

Newspapers 

Success - Next Level 
College Prep Exams 
Entries - Higher Level 
Formal Parent Input 

Mobility Rate 
Teacher Turnover 
Dropout Rate 
Awards 



100 79 50 

50 93 75 

83 43 50 

50 93 75 

40 64 100 

50 71 100 

17 50 25 

0 0 0 

17 50 25 

33 36 25 

33 29 25 

17 29 0 

17 43 0 

17 29 0 

17 14 0 

0 21 0 

0 14 50 

0 14 25 

0 14 25 

0 29 25 

17 7 0 

0 0 0 

0 0 25 

0 7 0 



Table 5: Percentages 


of Principals Who Use 


Sources 


By CAP Scores 






Relative 


to CAP Reading State Average 


Sources 


Below 


At 


Above 




n = 9 


n =^ 4 


n = 5 






% 




Sch/Class Observation 


100 


79 


50 


CAP 


78 


100 


80 


Norm Reference Tests 


/ o 




ou 


Informal Teacher Input 


50 


93 


75 


Informal Parent Input 


40 


64 


100 


Informal Student Input 


50 


71 


100 


Grades 


17 


50 


25 


Formal Teacher Input 


0 


4 3 


0 


Counselor Input 


17 


50 


25 


Discipline 


33 


36 


25 


Formal Student Input 


33 


29 


25 


Community Input 


17 


29 


0 


Superlnt/Other Adm Input 


1 7 


•t J 


0 


Attendance Rate 


17 


29 


0 


Proficiency Tests 


33 


0 


0 


Newspapers 


0 


21 


0 


Success - Next Level 


0 


14 


50 


College Prep Exams 


17 


14 


0 


Entries - Higher Level 


0 


14 


25 


Formal Parent Input 


0 


29 


25 


Mobility Rate 


17 


7 


0 


Teacher Turnover 


0 


0 


0 


Dropout Rate 


0 


0 


25 


Awards 


0 


7 


0 
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